feat(review): backtest-gated autonomous loosening of the satisfaction confidence floor#8163
Merged
Merged
Conversation
… confidence floor (#8121 narrow start) auto-tune.ts's OverridePayload states the historical rule: a loosening recommendation never carries a payload, because autonomous loosening is the regression risk the loop exists to avoid. The #8082 backtest primitives are exactly the missing risk measurement -- this ships the epic's approved narrow start for its named first candidate, LINKED_ISSUE_SATISFACTION_CONFIDENCE_FLOOR. evaluateSatisfactionFloorLoosening (pure) proposes the SMALLEST candidate step below the current floor that scores strictly improved on the visible split AND non-regressed on the deterministic held-out split (#8087's fixed-seed split, so repeated runs can never fish for a lucky partition), never below a hard 0.3 safety minimum, never on a small corpus. The run layer applies a cleared proposal by writing a system_flags override (the same operational-flag table the circuit breakers use) plus a calibration.satisfaction_floor_loosened audit event, with independent direction/bound re-checks at the write path. The live floor resolves inside runLoopOverLinkedIssueSatisfaction; the pure parse takes the floor as a defaulted parameter, byte-identical when absent. Everything is gated on SATISFACTION_FLOOR_AUTOTUNE_ENABLED (wrangler var, false by default): the override read, the apply path, and the manual trigger route POST /v1/internal/calibration/loosen-satisfaction-floor all 404/no-op until the operator opts in, and flipping the flag off instantly restores the shipped floor. 100% line+branch coverage on both new modules; floor threading covered in the pure and run suites. Advances #8121.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
This was referenced Jul 23, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LINKED_ISSUE_SATISFACTION_CONFIDENCE_FLOOR, a single scalar with real calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101 ground truth.auto-tune.ts's standing rule ("a loosening recommendation never carries a payload — autonomous loosening is the regression risk the loop exists to avoid") stays intact for everything else; this loop is the measured exception the Epic: verifiable pre-merge backtest for ORB/AMS deterministic rules #8082 backtest primitives exist to make safe.evaluateSatisfactionFloorLoosening(pure, 100% branch-covered) proposes the smallest candidate step below the current floor that scores strictlyimprovedon the visible split and non-regressedon the deterministic held-out split (calibration: deterministic seeded held-out/visible split of the backtest corpus #8087, fixed seed — repeated runs can never fish for a lucky partition), requires a real sample (≥20 visible / ≥5 held-out decided cases), and never proposes below a hard 0.3 safety minimum.runSatisfactionFloorLoosening): writes asystem_flagsoverride (the same operational-flag table the circuit breakers already use — no new storage) plus acalibration.satisfaction_floor_loosenedaudit event carrying both split comparisons, with independent direction/bound re-checks at the write path (a bogus proposal object cannot tighten or pass the minimum, tested via a mocked evaluator). One candidate step per run; re-runs evaluate from the current (possibly already-loosened) floor and can never oscillate upward.runLoopOverLinkedIssueSatisfactionresolves the live floor in one place; the pure parse takes it as a defaulted parameter — byte-identical behavior when absent. Everything (override read, apply path, and the manual triggerPOST /v1/internal/calibration/loosen-satisfaction-floor) is gated on the newSATISFACTION_FLOOR_AUTOTUNE_ENABLEDwrangler var,falseby default: the deploy is behavior-identical, and flipping the flag off instantly restores the shipped floor.Advances #8121 (the epic stays open for the decomposed remainder).
Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #123) — a linked open issue is required for every contributor PR.Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally;codecov/patchrequires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.npm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:testnpm run ui:buildnpm audit --audit-level=moderateIf any required check was skipped, explain why:
npm run test:ciran green end-to-end (1110 test files) on the pre-rebase head; the rebase onto feat(api): weekly rule-calibration trend endpoint over fired/override history and backtest runs #8156's merge resolved one textual conflict insrc/api/routes.ts(both branches append internal routes at the same anchor — both kept), re-verified with a cleantsc --noEmitplus both internal-route suites (14 tests). New-module coverage: 100% line AND branch on bothsatisfaction-floor-loosening.ts(6 tests) andsatisfaction-floor-loosening-run.ts(12 tests, incl. the mocked-evaluator defense-in-depth case and the best-effort audit failure path); floor threading covered in both the pure suite (3 new tests) and the run suite (3 new tests); route auth/flag-gate/response pinned in its own suite.cf-typegenregenerated and committed for the new var.Safety
UI Evidencesection below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.This is trust-critical by nature; the containment is layered: env flag off by default → evaluator's own gates → write-path direction/bound re-checks → validated read (a corrupted override row can never tighten the floor or pass the minimum, and reads fail safe to the shipped default) → every apply audited with full evidence. A stored-override read failure degrades to the shipped constant, never blocks the review pass. Negative paths tested: 401 without token, 404 with flag off, DB-error fail-safe.
UI Evidence
Not applicable — no UI change.
Notes
/v1/internal/decision/calibrationprecedent) is the manual trigger; calibration: schedule the satisfaction-floor loosening evaluation on the cron tick, with an apply alert #8158 schedules it.